Perceptual associative memory for a neuro-linguistic behavior recognition system

ABSTRACT

Techniques are disclosed for generating a syntax for a neuro-linguistic model of input data obtained from one or more sources. A stream of words of a dictionary built from a sequence of symbols are received. The symbols are generated from an ordered stream of normalized vectors generated from input data. Statistics for combinations of words co-occurring in the stream are evaluated. The statistics includes a frequency upon which the combinations of words co-occur. A model of combinations of words based on the evaluated statistics is updated. The model identifies statistically relevant words. A connected graph is generated. Each node in the connected graph represents one of the words in the stream. Edges connecting the nodes represent a probabilistic relationship between words in the stream. Phrases are identified based on the connected graph.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/569,161, filed Dec. 12, 2014, which is hereby incorporated byreference in its entirety.

BACKGROUND Field

Embodiments described herein generally relate to data analysis systems,and more particularly to building neuro-linguistic models of input dataobtained from one or more data sources.

Description of the Related Art

Many currently available surveillance and monitoring systems (e.g.,video surveillance systems, SCADA systems, and the like) are trained toobserve specific activities and alert an administrator after detectingthose activities.

However, such rules-based systems require advance knowledge of whatactions and/or objects to observe. The activities may be hard-coded intounderlying applications or the system may train itself based on anyprovided definitions or rules. In other words, unless the underlyingcode includes descriptions of certain behaviors or rules for generatingan alert for a given observation, the system is incapable of recognizingsuch behaviors. Such a rules-based approach is rigid. That is, unless agiven behavior conforms to a predefined rule, an occurrence of thebehavior can go undetected by the monitoring system. Even if the systemtrains itself to identify the behavior, the system requires rules to bedefined in advance for what to identify.

In addition, many surveillance systems, e.g., video surveillancesystems, require a significant amount of computing resources, includingprocessor power, storage, and bandwidth. For example, typical videosurveillance systems require a large amount of computing resources percamera feed because of the typical size of video data. Given the cost ofthe resources, such systems are difficult to scale.

SUMMARY

One embodiment presented herein includes a method for generating asyntax for a neuro-linguistic model of input data obtained from one ormore sources. The method generally includes receiving a stream of wordsof a dictionary built from a sequence of symbols. The symbols aregenerated from an ordered stream of normalized vectors generated frominput data. Statistics for combinations of words co-occurring in thestream are evaluated. The statistics includes a frequency upon which thecombinations of words co-occur. A model of combinations of words isupdated based on the evaluated statistics. The model identifiesstatistically relevant words. A connected graph is generated. Each nodein the connected graph represents one of the words in the stream. Edgesconnecting the nodes represent a probabilistic relationship betweenwords in the stream. Phrases are identified based on the connectedgraph.

Another embodiment presented herein includes a computer-readable storagemedium storing instructions, which, when executed on a processor,performs an operation generating a syntax for a neuro-linguistic modelof input data obtained from one or more sources. The operation itselfgenerally includes receiving a stream of words of a dictionary builtfrom a sequence of symbols. The symbols are generated from an orderedstream of normalized vectors generated from input data. Statistics forcombinations of words co-occurring in the stream are evaluated. Thestatistics includes a frequency upon which the combinations of wordsco-occur. A model of combinations of words is updated based on theevaluated statistics. The model identifies statistically relevant words.A connected graph is generated. Each node in the connected graphrepresents one of the words in the stream. Edges connecting the nodesrepresent a probabilistic relationship between words in the stream.Phrases are identified based on the connected graph.

Yet another embodiment presented herein includes a system having aprocessor and a memory storing one or more application programsconfigured to perform an operation for generating a syntax for aneuro-linguistic model of input data obtained from one or more sources.The operation itself generally includes receiving a stream of words of adictionary built from a sequence of symbols. The symbols are generatedfrom an ordered stream of normalized vectors generated from input data.Statistics for combinations of words co-occurring in the stream areevaluated. The statistics includes a frequency upon which thecombinations of words co-occur. A model of combinations of words isupdated based on the evaluated statistics. The model identifiesstatistically relevant words. A connected graph is generated. Each nodein the connected graph represents one of the words in the stream. Edgesconnecting the nodes represent a probabilistic relationship betweenwords in the stream. Phrases are identified based on the connectedgraph.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentdisclosure can be understood in detail, a more particular description ofthe disclosure, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlyexemplary embodiments and are therefore not to be considered limiting ofits scope, may admit to other equally effective embodiments.

FIG. 1 illustrates an example computing environment for aneuro-linguistic behavior recognition system, according to oneembodiment.

FIG. 2 illustrates a system architecture of a neuro-linguistic behaviorrecognition system, according to one embodiment.

FIG. 3 illustrates a method for collecting sensor data for use in aneuro-lingustic behavior recognition system, according to oneembodiment.

FIG. 4 illustrates a method for initializing a perceptual associativememory, according to one embodiment.

FIG. 5 illustrates a connected graph representing relationships betweenwords from a dictionary input into a perceptual associative memory,according to one embodiment.

FIG. 6 illustrates a method for sending a syntax to a cognitive module,according to one embodiment.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. It is contemplated that elements and features of oneembodiment may be beneficially incorporated in other embodiments withoutfurther recitation.

DETAILED DESCRIPTION

Embodiments presented herein describe a behavior recognition system. Thebehavior recognition system may be configured with one or more datacollector components that collect raw data values from different datasources (e.g., video data, building management data, SCADA data). Forexample, a behavior recognition system may be configured for videosurveillance. The behavior recognition system may include a datacollector component that retrieves video frames in real-time, separatesforeground objects from background objects, and tracks foregroundobjects from frame-to-frame. The data collector component may normalizethe video frame data into numerical values (e.g., falling within a rangefrom 0 to 1 with respect to a given data type).

In one embodiment, the behavior recognition system includes aneuro-linguistic module that performs neural network-based linguisticanalysis on the collected data. Specifically, for each type of datamonitored by a sensor, the neuro-linguistic module creates and refines alinguistic model of the normalized data. That is, the neuro-linguisticmodule builds a grammar used to describe the normalized data. Thelinguistic model includes symbols that serve as building blocks for thegrammar. The neuro-linguistic module identifies combinations of symbolsto build a dictionary of words. Once the dictionary is built, theneuro-linguistic module identifies phrases that include variouscombinations of words in the dictionary. The behavior recognition systemuses such a linguistic model to describe what is being observed. Thelinguistic model allows the behavior recognition system to distinguishbetween normal and abnormal activity observed in the input data. As aresult, the behavior recognition system can issue alerts wheneverabnormal activity occurs.

To generate the linguistic model, a neuro-linguistic module receivesnormalized data values and organizes the data into clusters. Theneuro-linguistic module evaluates statistics of each cluster andidentifies statistically relevant clusters. Further, theneuro-linguistic module generates symbols, e.g., letters, correspondingto each statistically relevant cluster. Thus, input values mapping to agiven cluster may correspond to a symbol.

The neuro-linguistic module generates a lexicon, i.e., builds adictionary, of observed combinations of symbols, i.e., words, based on astatistical distribution of symbols identified in the input data.Specifically, the neuro-linguistic module may identify patterns ofsymbols in the input data at different frequencies of occurrence.Further, the neuro-linguistic module can identify statistically relevantcombinations of symbols at different lengths (e.g., from one-symbol to amaximum-symbol word length). The neuro-linguistic module may includesuch statistically relevant combinations of symbols in a dictionary usedto identify phrases for the linguistic model.

Using words from the dictionary, the neuro-linguistic module generatesphrases based on probabilistic relationships of each word occurring insequence relative to other words as additional data is observed. Forexample, the neuro-linguistic module may identify a relationship betweena given three-letter word that frequently appears in sequence with agiven four-letter word, and so on. The neuro-linguistic moduledetermines a syntax based on the identified phrases.

The syntax allows the behavior recognition system to learn, identify,and recognize patterns of behavior without the aid or guidance ofpredefined activities. Unlike a rules-based surveillance system, whichcontains predefined patterns of what to identify or observe, thebehavior recognition system learns patterns by generalizing input andbuilding behavior memories of what is observed. Over time, the behaviorrecognition system uses these memories to distinguish between normal andanomalous behavior reflected in observed data.

For instance, the neuro-linguistic module builds letters, words,phrases, and estimates an “unusualness score” for each identifiedletter, word, or phrase. The unusualness score (for a letter, word, orphrase observed in input data) provides a measure of how infrequentlythe letter, word, or phrase has occurred relative to past observations.Thus, the behavior recognition system may use the unusualness scores toboth measure how unusual a current syntax is, relative to a stable modelof symbols (i.e., letters), a stable model of words built from thesymbols (i.e., a dictionary) and a stable model of phrase built from thewords (i.e., a syntax)—collectively the neuro-linguistic model.

As the neuro-linguistic module continues to receive input data, theneuro-linguistic module may decay, reinforce, and generate the letters,words, and syntax models. In parlance with the machine learning field,the neuro-linguistic module “learns on-line” as new data is received andoccurrences a given type of input data either increases, decreases,appears, or disappears.

FIG. 1 illustrates components of a behavioral recognition system 100,according to one embodiment. As shown, the behavioral recognition system100 includes one or more input source devices 105, a network 110, andone or more computer systems 115. The network 110 may transmit datainput by the source devices 105 to the computer system 115. Generally,the computing environment 100 may include one or more physical computersystems 115 connected via a network (e.g., the Internet). Alternatively,the computer systems 115 may be cloud computing resources connected bythe network. Illustratively, the computer system 115 includes one ormore central processing units (CPU) 120, one or more graphics processingunits (GPU) 121, network and I/O interfaces 122, a storage 124 (e.g., adisk drive, optical disk drive, and the like), and a memory 123 thatincludes a sensor management module 130, a sensory memory component 135,and a machine learning engine 140. The storage 124 includes a modelrepository 145.

The CPU 120 retrieves and executes programming instructions stored inthe memory 123 as well as stores and retrieves application data residingin the storage 124. In one embodiment, the GPU 121 implements a ComputeUnified Device Architecture (CUDA). Further, the GPU 121 is configuredto provide general purpose processing using the parallel throughputarchitecture of the GPU 121 to more efficiently retrieve and executeprogramming instructions stored in the memory 123 and also to store andretrieve application data residing in the storage 124. The parallelthroughput architecture provides thousands of cores for processing theapplication and input data. As a result, the GPU 121 leverages thethousands of cores to perform read and write operations in a massivelyparallel fashion. Taking advantage of the parallel computing elements ofthe GPU 121 allows the behavior recognition system 100 to better processlarge amounts of incoming data (e.g., input from a video and/or audiosource). As a result, the behavior recognition system 100 may scale withrelatively less difficulty.

The sensor management module 130 provides one or more data collectorcomponents. Each of the collector components is associated with aparticular input data source, e.g., a video source, a SCADA (supervisorycontrol and data acquisition) source, an audio source, etc. Thecollector components retrieve (or receive, depending on the sensor)input data from each source at specified intervals (e.g., once a minute,once every thirty minutes, once every thirty seconds, etc.). The sensormanagement module 130 controls the communications between the datasources. Further, the sensor management module 130 normalizes input dataand sends the normalized data to the sensory memory component 135.

The sensory memory component 135 is a data store that transfers largevolumes of data from the sensor management module 130 to the machinelearning engine 140. The sensory memory component 135 stores the data asrecords. Each record may include an identifier, a timestamp, and a datapayload. Further, the sensory memory component 135 aggregates incomingdata in a time-sorted fashion. Storing incoming data from each of thedata collector components in a single location where the data may beaggregated allows the machine learning engine 140 to process the dataefficiently. Further, the computer system 115 may reference data storedin the sensory memory component 135 in generating alerts for anomalousactivity. In one embodiment, the sensory memory component 135 may beimplemented in via a virtual memory file system in the memory 123. Inanother embodiment, the sensory memory component 135 is implementedusing a key-value share.

The machine learning engine 140 receives data output from the sensormanagement module 135. Generally, components of the machine learningengine 140 generate a linguistic representation of the normalizedvectors. As described further below, to do so, the machine learningengine 140 clusters normalized values having similar features andassigns a distinct symbol to each cluster. The machine learning engine140 may then identify recurring combinations of symbols (i.e., words) inthe data. The machine learning engine 140 then similarly identifiesrecurring combinations of words (i.e., phrases) in the data.

Note, however, FIG. 1 illustrates merely one possible arrangement of thebehavior recognition system 100. For example, although the input datasources 105 are shown connected to the computer system 115 via network110, the network 110 is not always present or needed (e.g., an inputsource such as a video camera may be directly connected to the computersystem 115).

FIG. 2 illustrates a system architecture of the behavior recognitionsystem, according to one embodiment. As shown, the sensor managementmodule 130 and the machine learning engine 140 communicate via apersistence layer 210.

The persistence layer 210 includes data stores that maintain informationused by components of the computer system 115. For example, thepersistence layer 210 includes data stores that maintain informationdescribing properties of the data collector modules 202, systemproperties (e.g., serial numbers, available memory, available capacity,etc. of the computer system 115), and properties of the source driver(e.g., active plug-ins 118, active sensors associated with each datasource, normalization settings, etc.). Other data stores may maintainlearning model information, system events, and behavioral alerts. Inaddition, the sensory memory component 135 resides in the persistencelayer 210.

The machine learning engine 140 itself includes a neuro-linguisticmodule 215 and a cognitive module 225. The neuro-linguistic module 215performs neural network-based linguistic analysis of normalized inputdata to build a neuro-linguistic model of the observed input data. Thebehavior recognition system can use the linguistic model to describesubsequently observed activity. However, rather than describing theactivity based on predefined objects and actions, the neuro-linguisticmodule 215 develops a custom language based on symbols, words, andphrases generated from the input data. As shown, the neuro-linguisticmodule 215 includes a data transactional memory (DTM) component 216, aclassification analyzer component 217, a mapper component 218, a lexicalanalyzer component 219, and a perceptual associative memory (PAM)component 220.

In one embodiment, the DTM component 216 retrieves the normalizedvectors of input data from the sensory memory component 135 and stagesthe input data in the pipeline architecture provided by the GPU 121. Theclassification analyzer component 217 evaluates the normalized dataorganized by the DTM component 216 and maps the data on a neuralnetwork. In one embodiment, the neural network is a combination of aself-organizing map (SOM) and an adaptive resonance theory (ART)network.

The mapper component 218 clusters the data streams based on valuesoccurring repeatedly in association with one another. Further, themapper component 218 generates a set of clusters for each input feature.For example, assuming that the input data corresponds to video data,features may include location, velocity, acceleration etc. The mappercomponent 218 would generate separate sets of clusters for each of thesefeatures. The mapper component 218 identifies symbols (i.e., builds analphabet of letters) based on the clustered input data. Specifically,the mapper component 218 determines a statistical distribution of datain each cluster. For instance, the mapper component 218 determines amean, variance, and standard deviation for the distribution of values inthe cluster. The mapper component 218 also updates the statistics asmore normalized data is received. Further, each cluster may beassociated with a statistical significance score. The statisticalsignificance for a given cluster increases as more data is receivedwhich maps to that cluster. In addition, the mapper component 218 decaysthe statistical significance of the cluster as the mapper component 218observes data mapping to the cluster less often over time.

In one embodiment, the mapper component 218 assigns a set of symbols toclusters having statistical significance. A cluster may have statisticalsignificance if a threshold amount of input data mapping to that clusteris exceeded. A symbol may be described as a letter of an alphabet usedto create words used in the neuro-linguistic analysis of the input data.A symbol provides a “fuzzy” representation of the data belonging to agiven cluster.

Further, the mapper component 218 is adaptive. That is, the mappercomponent 218 may identify new symbols corresponding to new clustersgenerated from the normalized data, as such clusters are reinforced overtime (resulting in such clusters reaching a level statisticalsignificance relative to the other clusters that emerge from the inputdata). The mapper component 218 “learns on-line” and may merge similarobservations to a more generalized cluster. The mapper component 218 mayassign a distinct symbol to the resulting cluster.

Once a cluster has reached statistical significance (i.e., data observedas mapping to that cluster has reached a threshold amount of points),the mapper component 219 begins sending corresponding symbols to thelexical analyzer component 219 in response to normalized data that mapsto that cluster. In one embodiment, the mapper component 218 limitssymbols that can be sent to the lexical component 219 to the moststatistically significant clusters. In practice, outputting symbols(i.e., letters) assigned to the top thirty-two clusters has shown to beeffective. However, other amounts may also prove effective, such as thetop sixty-four or 128 most frequently recurring clusters. Note, overtime, the most frequently observed symbols may change as clustersincrease (or decrease) in statistical significance. As such, it ispossible for a given cluster to lose statistical significance. Overtime, thresholds for statistical significance can increase, and thus, ifthe amount of observed data mapping to a given cluster fails to meet athreshold, then the cluster loses statistical significance.

In one embodiment, the mapper component 218 evaluates an unusualnessscore for each symbol. The unusualness score is based on the frequencyof a given symbol relative to other symbols observed in the input datastream, over time. The unusualness score may increase or decrease overtime as the neuro-linguistic module 215 receives additional data.

The mapper component 218 sends a stream of the symbols (e.g., letters),timestamp data, unusualness scores, and statistical data (e.g., arepresentation of the cluster associated with a given symbol) to thelexical analyzer component 219. The lexical analyzer component 219builds a dictionary based on symbols output from the mapper component218. In practice, the mapper component 218 may need approximately 5000observations (i.e., normalized vectors of input data) to generate astable alphabet of symbols.

The lexical analyzer component 219 builds a dictionary that includescombinations of co-occurring symbols, e.g., words, from the symbolstransmitted by the mapper component 218. The lexical analyzer component219 identifies repeating co-occurrences of letters and features outputfrom the mapper component 218 and calculates frequencies of theco-occurrences occurring throughout the symbol stream. The combinationsof symbols may represent a particular activity, event, etc.

In one embodiment, the lexical analyzer component 219 limits the lengthof words in the dictionary to allow the lexical analyzer component 219to identify a number of possible combinations without adverselyaffecting the performance of the computer system 115. Further, thelexical analyzer component 219 may use level-based learning models toanalyze symbol combinations and learn words. The lexical analyzercomponent 219 learns words up through a maximum symbol combinationlength at incremental levels, i.e., where one-letter words are learnedat a first level, two-letter words are learned at a second level, and soon. In practice, limiting a word to a maximum of five or six symbols hasshown to be effective.

Like the mapper component 218, the lexical analyzer component 219 isadaptive. That is, the lexical analyzer component 219 may learn andgenerate words in the dictionary over time. The lexical analyzercomponent 219 may also reinforce or decay the statistical significanceof words in the dictionary as the lexical analyzer component 219receives subsequent streams of symbols over time. Further, the lexicalanalyzer component 219 may determine an unusualness score for each wordbased on how frequently the word recurs in the data. The unusualnessscore may increase or decrease over time as the neuro-linguistic module215 processes additional data.

In addition, as additional observations (i.e., symbols) are passed tothe lexical analyzer component 219 and identified as a being part of agiven word, the lexical analyzer component 219 may determine that theword model has matured. Once a word model has matured, the lexicalanalyzer component 219 may output observations of those words in themodel to the PAM component 219. In one embodiment, the lexical analyzercomponent 219 limits words sent to the PAM component 320 to the moststatistically relevant words. In practice, for each single sample,outputting occurrences of the top thirty-two most frequently occurringwords has shown to be effective (while the most frequently occurringwords stored in the models can amount to thousands of words). Note, overtime, the most frequently observed words may change as the observationsof incoming letters change in frequency (or as new letters emerge by theclustering of input data by the mapper component 218.

Once the lexical analyzer component 219 has built the dictionary (i.e.,identifies words that have a reached a predefined statisticalsignificance), the lexical analyzer component 219 sends occurrences ofwords subsequently observed in the input stream to the PAM component220. The PAM component 220 builds a syntax of phrases with from thewords output by the lexical analyzer component 219. In practice, lexicalanalyzer component 219 may build a useful dictionary of words afterreceiving approximately 15,000 observations (i.e., input letters fromthe mapper component 218).

The PAM component 220 identifies a syntax of phrases based on thesequence of words output from the lexical analyzer component 219.Specifically, the PAM component 220 receives the words identified by thelexical analyzer component 219 generates a connected graph, where thenodes of the graph represent the words, and the edges represent arelationship between the words. The PAM component 220 may reinforce ordecay the links based on the frequency that the words are connected withone another in a data stream.

Similar to the mapper component 218 and the lexical analyzer component219, the PAM component 220 determines an unusualness score for eachidentified phrase based on how frequently the phrase recurs in thelinguistic data. The unusualness score may increase or decrease overtime as the neuro-linguistic module 215 processes additional data.

Similar to the lexical analyzer component 219, the PAM component 220 maylimit the length of a given phrase to allow the PAM component 220 to beable to identify a number of possible combinations without adverselyaffecting the performance of the computer system 115.

The PAM component 220 identifies syntax phrases over observations ofwords output from the lexical analyzer component 219. As observations ofwords accumulate, the PAM component 220 may determine that a givenphrase has matured, i.e., a phrase has reached a measure of statisticalrelevance. The PAM component 220 then outputs observations of thatphrase to the cognitive module 225. The PAM component 220 sends datathat includes a stream of the symbols, words, phrases, timestamp data,unusualness scores, and statistical calculations to the cognitive module325. In practice, the PAM component 220 may obtain a meaningful set ofphrases after observing about 5000 words from the lexical analyzercomponent 219.

After maturing, the generated letters, words, and phrases form a stableneuro-linguistic model of the input data that the computer system 115uses to compare subsequent observations of letters, words, and phrasesagainst the stable model. The neuro-linguistic module 215 updates thelinguistic model as new data is received. Further, the neuro-linguisticmodule 215 may compare a currently observed syntax to the model. Thatis, after building a stable set of letters, the neuro-linguistic module215 may build a stable model of words (e.g., a dictionary). In turn, theneuro-linguistic module 215 may be used to build a stable model ofphrases (e.g., a syntax). Thereafter, when the neuro-linguistic module215 receives subsequently normalized data, the module 215 can output anordered stream of symbols, words, and phrases, all of which can becompared to the stable model to identify interesting patterns or detectdeviations occurring in the stream of input data.

As shown, the cognitive module 226 includes a workspace 226, a semanticmemory 230, codelet templates 235, episodic memory 240, long term memory245, and an anomaly detection component 250. The semantic memory 230stores the stable neuro-linguistic model described above, i.e., a stablecopy from the mapper component 218, lexical analyzer component 219, andthe PAM component 220.

In one embodiment, the workspace 226 provides a computational engine forthe machine learning engine 140. The workspace 226 performs computations(e.g., anomaly modeling computations) and stores immediate results fromthe computations.

The workspace 226 retrieves the neuro-linguistic data from the PAMcomponent 220 and disseminates this data to different portions of thecognitive module 225 as needed.

The episodic memory 240 stores linguistic observations related to aparticular episode in the immediate past and may encode specificdetails, such as the “what” and the “when” of a particular event.

The long-term memory 245 stores generalizations of the linguistic datawith particular episodic details stripped away. In this way, when a newobservation occurs, memories from the episodic memory 240 and thelong-term memory 245 may be used to relate and understand a currentevent, i.e., the new event may be compared with past experience (asrepresented by previously observed linguistic data), leading to bothreinforcement, decay, and adjustments to the information stored in thelong-term memory 245, over time. In a particular embodiment, thelong-term memory 245 may be implemented as an ART network and asparse-distributed memory data structure. Importantly, however, thisapproach does not require events to be defined in advance.

The codelet templates 235 provide a collection of executable codelets,or small pieces of code that evaluate different sequences of events todetermine how one sequence may follow (or otherwise relate to) anothersequence. The codelet templates 325 may include deterministic codeletsand stochastic codelets. More generally, a codelet may detectinteresting patterns from the linguistic representation of input data.For instance, a codelet may compare a current observation (i.e., acurrent phrase instance with what has been observed in the past) withpreviously observed activity stored in the semantic memory 230. Byrepeatedly scheduling codelets for execution, copying memories andpercepts to/from the workspace 226, the cognitive module 225 performs acognitive cycle used to observe, and learn, about patterns of behaviorthat occur within the linguistic data.

The anomaly detection component 250 evaluates unusualness scores sent bythe neuro-linguistic module 215 to determine whether to issue an alertin response to some abnormal activity indicated by the unusualnessscores. Specifically, the anomaly detection component 250 is providesprobabilistic histogram models (e.g., an unusual lexicon score model, anunusual syntax score model, and an anomaly model) which represent theunusualness scores. The unusual lexicon score model and unusual syntaxscore model are generated based on unusualness scores sent from thelexical analyzer component 219 and the PAM component 220. The anomalymodel receives input percentiles from the unusual lexicon score modeland unusual syntax score model and generates an absolute unusualnessscore based on the percentiles. The anomaly detection component 250evaluates the scores and determines whether to send an alert based on agiven score. The anomaly detection component 250 may send alert data toan output device, where an administrator may view the alert, e.g., via amanagement console.

The cognitive module 225 performs learning analysis on the linguisticcontent delivered to semantic memory 230 (i.e., the identified symbols,words, phrases) by comparing new observations to the learned patterns inthe stable neuro-linguistic model kept in semantic memory 230 and thenestimating the rareness of these new observations.

Specifically, the anomaly detection component 250 evaluates theunusualness scores of each of the symbols, words, and phrases toidentify abnormal occurrences in the observed data. Once an anomalousobservation has been identified, the anomaly component may issue analert (e.g., notify an administrator or user of the computer system115).

FIG. 3 illustrates a method 300 for collecting sensor data for use in aneuro-lingustic behavior recognition system, according to oneembodiment. More specifically, method 300 describes a method for a datacollector to retrieve data from an associated input device and send thedata to the neuro-linguistic module 215. For this example, assume that adata collector module 202 is a video source capturing image data at agiven frame rate. Of course, a variety of data collector components 202can be used.

Method 300 begins at step 305, where the data collector module 202retrieves (or receives) data from the source input device. In this case,the data collector module 202 may retrieve video frames from a videosource, such as a video camera positioned to observe a particularlocation, such as a hotel lobby. Further, the data collector module 202identifies data values to send to the sensory memory component 135. Todo so, the data collector module 202 may evaluate the video frames toseparate foreground objects from background objects, measure appearanceand kinematic information of the identified foreground objects, andtrack foreground objects moving across the scene (i.e., the field ofview of the camera). As a result, the data collector module 202generates a set of data values characterizing appearance and kinematicaspects of the objects depicted in video frames.

At step 310, the data collector module 202 normalizes each data value toa numerical value falling within a range, e.g., between 0 to 1,inclusive, relative to the type of that data value. For example, valuesassociated with kinematic features are normalized from 0 to 1 relativeto other values associated with kinematic features. Doing so convertseach value to a common format and allows the neuro-linguistic module 215to recognize recurring events in the video stream.

After normalizing the values, at step 315, the data collector module 202identifies additional data associated with the normalized values, suchas a timestamp of a given value, an average associated with the datatype (e.g., kinematic features, appearance features, location, position,etc.) of the value, and historical high and low values for that datatype. Doing so allows the data collector module 202 to readjust thenormalization in the event that the video source is modified.Specifically, the data collector module 202 references the identifiedhistorical values and averages to readjust the normalization.

At step 320, the data collector module 202 sends a vector of thenormalized values and associated data to the sensory memory component135. As stated, the sensory memory component 135 stores the normalizedvalues and associated data. The neuro-linguistic module 215 may thenretrieve the normalized values from the sensory memory component 135 andperform linguistic analysis thereafter.

FIG. 4 illustrates a method 400 for initializing the PAM component 220,according to one embodiment. Method 400 begins at step 405, where thePAM component 220 receives a stream of observations of words from thelexical analyzer component 219. As stated, the lexical analyzercomponent 219 generates the words by evaluating the statisticalfrequency of groups of symbols as output by the mapper component 218. Asgroups of symbols (again words) appear, the lexical analyzer component219 identifies statistically relevant combinations of symbols, i.e.,words, and outputs the words to the PAM component 220. As stated, thesymbols, words, and phrases are identified to build a statistical modelof the frequency for the symbols, words, and phrases as commonlyobserved, i.e., statistically relevant, in the stream.

At step 410, the PAM component 220 evaluates statistics of theco-occurring words. Such statistics may include how often the PAMcomponent 220 observes each word co-occurring with the other. In oneembodiment, the statistics may include a statistical significance scoredetermined by how frequently a given word co-occurs with another wordpassed in by the lexical analyzer component 219. To determine howfrequently each word is observed with one another, the PAM component 220may generate a matrix that arranges each of the words based onco-occurrence.

At step 420, the PAM component 220 generates a connected graph. Eachnode in the connected graph represents one of the words output by thelexical component 219. Undirected edges connect nodes to indicate arelationship between the words that co-occur with a relative frequency.Further, the edges may be weighted based on the statistical significancescore between words. Strings of connected nodes may represent phrasesthat the behavior recognition system uses to describe data input to theneuro-linguistic module 210, clustered into symbols (by the mappercomponent 217) and parsed into words (by the lexical component).

Over time, the PAM component 220 can receive additional observations ofwords from the lexical analyzer component 219. The PAM component 220 mayreinforce a given edge as the statistical significance score increases,e.g., as the PAM component 220 observes subsequent instances of twowords co-occurring with one another. In addition, the PAM component 220may eliminate the edge as fewer observations occur over time (i.e., asthe statistical significance score decreases over time).

FIG. 5 illustrates an example connected graph 500 generated by the PAMcomponent 220, according to one embodiment. When the lexical analyzercomponent 219 initially outputs a stream of words to the PAM component220, the PAM component 220 generates the connected graph 500.

The nodes (represented by the circles) represent identified words sentby the lexical analyzer component 219. The undirected edges connectingthe nodes represent that the PAM component 220 has observed theconnected words to co-occur in the stream of words. Further, the edgesmay be weighted based on a statistical significance score between twoco-occurring words. In addition, the PAM component 220 may reinforce ordecay the edges based as the statistical significance score increases ordecreases, respectively. For example, the statistical significance scoremay increase if the PAM component 220 observes further co-occurrencesbetween two words. The PAM component 220 can reinforce the edgeconnecting both words in the graph as a result of the increase.

The graph 500 establishes a syntax of phrases identified by the PAMcomponent 220. A phrase is a sequence of words that may occur in theword stream based on probabilistic relationships of each word appearingin the order of a given phrase (a given path through the connectedgraph). The PAM component 220 outputs phrases observed in the stream ofwords output by the mapper having co-occurring words with a statisticalsignificance score exceeding a given threshold. Further, the PAMcomponent 220 may send metadata characterizing phrases output by the PAMcomponent 220 to the cognitive module 225. For example, the PAMcomponent 200 may include metadata from the mapper component 218 and thelexical analyzer component 219, such as symbols, words, and associatedstatistics sent by the mapper component 218 and the lexical analyzercomponent 219. Collectively, the symbols, words, phrases, along with allthe metadata related to the frequencies at which the symbols, words, andphrases are observed and the scoring assigned by the mapper and lexicalanalyzer components form a neuro-linguistic model describing the inputdata. In turn, the cognitive module 225 analyzes the neuro-linguisticmodel to interpret incoming data, e.g., in identifying normal andabnormal behavior.

FIG. 6 illustrates a method 600 for sending phrases observed by the PAMcomponent (in a word-stream supplied by the lexical component) to thecognitive module 225, according to one embodiment. Specifically, method600 describes how the PAM component 220 processes incoming words fromthe lexical analyzer component 219 after initialization, i.e., afterinitially generating the fully connected graph. Method 600 begins atstep 605, where the PAM component 220 begins receiving a stream ofobservations of words from the lexical analyzer component 219. The PAMcomponent 220 observes co-occurrences between the words received, e.g.,a word A is situated next to a word B in the stream. The PAM component220 may also identify newly co-occurring words with the input.

At step 610, the PAM component 220 updates co-occurrence statisticsassociated with each observation in the stream. For instance, the PAMcomponent 220 may increase a statistical significance score for a pairof co-occurring words if observed in the stream. As another example, apair of words previously not observed to co-occur may be observed toco-occur in the stream. If so, the PAM component 220 increases astatistical significance score for the pair of words.

At step 615, the PAM component 220 updates the fully connected graph.For example, the PAM component 220 may reinforce edges at portions ofthe graph where the statistical significance score increased. Further,the PAM component 220 may add edges to observed pairs that have reachedstatistical relevance (e.g., the statistical significance score hasexceeded a threshold).

At step 620, the PAM component 220 outputs phrases with co-occurringwords having a statistical significance score that exceeds a giventhreshold. Exceeding such a threshold may indicate that a given phraseis matured and may be sent to the cognitive module 225. In addition, thePAM component 220 may also send symbols, words, and associatedstatistics to accompany the phases. As stated, such symbols, words, andphrases form a neuro-linguistic model that the behavior recognitionsystems uses to describe input data.

One embodiment of the present disclosure is implemented as a programproduct for use with a computer system. The program(s) of the programproduct defines functions of the embodiments (including the methodsdescribed herein) and can be contained on a variety of computer-readablestorage media. Examples of computer-readable storage media include (i)non-writable storage media (e.g., read-only memory devices within acomputer such as CD-ROM or DVD-ROM disks readable by an optical mediadrive) on which information is permanently stored; (ii) writable storagemedia (e.g., floppy disks within a diskette drive or hard-disk drive) onwhich alterable information is stored. Such computer-readable storagemedia, when carrying computer-readable instructions that direct thefunctions of the present disclosure, are embodiments of the presentdisclosure. Other examples media include communications media throughwhich information is conveyed to a computer, such as through a computeror telephone network, including wireless communications networks.

In general, the routines executed to implement the embodiments of thepresent disclosure may be part of an operating system or a specificapplication, component, program, module, object, or sequence ofinstructions. The computer program of the present disclosure iscomprised typically of a multitude of instructions that will betranslated by the native computer into a machine-readable format andhence executable instructions. Also, programs are comprised of variablesand data structures that either reside locally to the program or arefound in memory or on storage devices. In addition, various programsdescribed herein may be identified based upon the application for whichthey are implemented in a specific embodiment of the disclosure.However, it should be appreciated that any particular programnomenclature that follows is used merely for convenience, and thus thepresent disclosure should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

As described, embodiments herein provide techniques for determining asyntax based on a dictionary of words that represents data input from asource (e.g., video source, SCADA source, network security source, etc.)via a neuro-linguistic behavior recognition system. The symbols, words,and syntax form the basis for a linguistic model used to describe inputdata observed by the behavior recognition system. The behaviorrecognition system analyzes and learns behavior based on the linguisticmodel to distinguish between normal and abnormal activity in observeddata. Advantageously, this approach does not relying on predefinedpatterns to identify behaviors and anomalies but instead learns patternsand behaviors by observing a scene and generating information on what itobserves.

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

1. (canceled)
 2. A method, comprising: receiving a plurality of videoframes from a video source, the plurality of video frames including arepresentation of an object; normalizing, via at least one processor,data in each video frame from the plurality of video frames to obtainnormalized data for the plurality of video frames; generating, via theat least one processor, a syntax for a neuro-linguistic model, thesyntax including a stable model of phrases; identifying, via the atleast one processor, instances of at least one machine-readable phrasefrom the stable model of phrases and based on a connected graph;calculating, via the at least one processor, an unusualness score for anobservation of a first machine-readable phrase from the at least onemachine-readable phrase identified in the connected graph; andpublishing, via the at least one processor, an alert associated with theobservation of the first machine-readable phrase, the alert indicatingan anomaly in the behavior of the object.
 3. The computer-implementedmethod of claim 2, wherein generating the syntax includes: generating,via the at least one processor, a plurality of machine-readable symbolsfrom the normalized data, each machine-readable symbol from theplurality of machine-readable symbols being associated with a distinctcluster of the normalized data, and building a dictionary ofmachine-readable words based on an observed sequence of the plurality ofmachine-readable symbols.
 4. The computer-implemented method of claim 2,wherein generating the syntax includes: evaluating, via the at least oneprocessor, statistics for a plurality of combinations ofmachine-readable words co-occurring in a stream of machine-readablewords, the statistics including a frequency at which combinations ofmachine-readable words from the plurality of combinations ofmachine-readable words co-occur; and updating, via the at least oneprocessor, a model of combinations of machine-readable words based onthe evaluated statistics, the model identifying statistically relevantobservations of co-occurring machine-readable words.
 5. Thecomputer-implemented method of claim 2, wherein generating the syntaxincludes generating, via the at least one processor, a connected graphhaving a plurality of nodes and a plurality of edges, each node from theplurality of nodes representing one machine-readable word in a stream ofmachine-readable words, and each edge from the plurality of edgesconnecting the nodes representing a probabilistic measure ofco-occurrence of pairs of statistically relevant words in the stream ofmachine-readable words.
 6. The computer-implemented method of claim 2,wherein generating the syntax includes generating a connected graph, viathe at least one processor, the connected graph including a plurality ofnodes and a plurality of edges, each edge from the plurality of edgesweighted based on a statistical significance score between each pair ofnodes from the plurality of nodes.
 7. The computer-implemented method ofclaim 2, further comprising: receiving a stream of machine-readablewords; and updating statistics of the co-occurring words based on thestream of machine-readable words.
 8. The computer-implemented method ofclaim 2, wherein generating the syntax includes generating a dictionaryof machine-readable words based on an observed-sequence of themachine-readable symbols, the method further comprising at least one of:decreasing a statistical significance score of co-occurringmachine-readable words, from the dictionary of machine-readable words,that are less frequently observed over time; or increasing a statisticalsignificance score of co-occurring machine-readable words, from thedictionary of machine-readable words, that are more frequently observedover time.
 9. A non-transitory computer-readable storage medium storinginstructions that, when executed by a processor, cause the processor to:receive a plurality of video frames from a video source, the pluralityof video frames including a representation of an object; normalize, viaat least one processor, data in each video frame from the receivedplurality of video frames to obtain normalized data for the plurality ofvideo frames; generate, via the at least one processor, a syntax for aneuro-linguistic model, the syntax including a stable model of phrases;identify, via the at least one processor, instances of at least onemachine-readable phrase from the stable model of phrases and based on aconnected graph; calculate, via the at least one processor, anunusualness score for an observation of a first machine-readable phrasefrom the at least one machine-readable phrase identified in theconnected graph; and publish, via the at least one processor, an alertassociated with the observation of the first machine-readable phrase,the alert indicating an anomaly in the behavior of the object.
 10. Thenon-transitory computer-readable storage medium of claim 9, wherein theinstructions to generate the syntax include instructions to: generate aplurality of machine-readable symbols from the normalized data, eachmachine-readable symbol from the plurality of machine-readable symbolsbeing associated with a distinct cluster of the normalized data, andbuild a dictionary of machine-readable words based on an observedsequence of the plurality of machine-readable symbols.
 11. Thenon-transitory computer-readable storage medium of claim 9, wherein theinstructions to generate the syntax include instructions to: evaluatestatistics for a plurality of combinations of machine-readable wordsco-occurring in a stream of machine-readable words, the statisticsincluding a frequency at which combinations of machine-readable wordsfrom the plurality of combinations of machine-readable words co-occur;and update a model of combinations of machine-readable words based onthe evaluated statistics, the model identifying statistically relevantobservations of co-occurring machine-readable words.
 12. Thenon-transitory computer-readable storage medium of claim 9, wherein theinstructions to generate the syntax include instructions to generate aconnected graph having a plurality of nodes and a plurality of edges,each node from the plurality of nodes representing one machine-readableword in a stream of machine-readable words, and each edge from theplurality of edges connecting the nodes representing a probabilisticmeasure of co-occurrence of pairs of statistically relevant words in thestream of machine-readable words.
 13. The non-transitorycomputer-readable storage medium of claim 9, wherein the instructions togenerate the syntax include instructions to generate a connected graph,the connected graph including a plurality of nodes and a plurality ofedges, each edge from the plurality of edges weighted based on astatistical significance score between each pair of nodes from theplurality of nodes.
 14. The non-transitory computer-readable storagemedium of claim 9, further storing instructions that, when executed bythe processor, cause the processor to: receive a stream ofmachine-readable words; and update statistics of the co-occurring wordsbased on the stream of machine-readable words.
 15. The non-transitorycomputer-readable storage medium of claim 9, wherein the instructions togenerate the syntax include instructions to generate a dictionary ofmachine-readable words based on an observed-sequence of themachine-readable symbols, the non-transitory computer-readable storagemedium further storing instructions that, when executed by theprocessor, cause the processor to: decrease a statistical significancescore of co-occurring machine-readable words, from the dictionary ofmachine-readable words, that are less frequently observed over time; orincrease a statistical significance score of co-occurringmachine-readable words, from the dictionary of machine-readable words,that are more frequently observed over time.
 16. A system, comprising: aprocessor; and a memory storing processor-executable instructions that,when executed by the processor, cause the processor to: receive aplurality of video frames from a video source, the plurality of videoframes including a representation of an object; normalize, via at leastone processor, data in each video frame from the received plurality ofvideo frames to obtain normalized data for the plurality of videoframes; generate, via the at least one processor, a syntax for aneuro-linguistic model, the syntax including a stable model of phrases;identify, via the at least one processor, instances of at least onemachine-readable phrase from the stable model of phrases and based on aconnected graph; calculate, via the at least one processor, anunusualness score for an observation of a first machine-readable phrasefrom the at least one machine-readable phrase identified in theconnected graph; and publish, via the at least one processor, an alertassociated with the observation of the first machine-readable phrase,the alert indicating an anomaly in the behavior of the object.
 17. Thesystem of claim 16, wherein the instructions to generate the syntaxinclude instructions to: generate a plurality of machine-readablesymbols from the normalized data, each machine-readable symbol from theplurality of machine-readable symbols being associated with a distinctcluster of the normalized data, and build a dictionary ofmachine-readable words based on an observed sequence of the plurality ofmachine-readable symbols.
 18. The system of claim 16, wherein theinstructions to generate the syntax include instructions to: evaluatestatistics for a plurality of combinations of machine-readable wordsco-occurring in a stream of machine-readable words, the statisticsincluding a frequency at which combinations of machine-readable wordsfrom the plurality of combinations of machine-readable words co-occur;and update a model of combinations of machine-readable words based onthe evaluated statistics, the model identifying statistically relevantobservations of co-occurring machine-readable words.
 19. The system ofclaim 16, wherein the instructions to generate the syntax includeinstructions to generate a connected graph having a plurality of nodesand a plurality of edges, each node from the plurality of nodesrepresenting one machine-readable word in a stream of machine-readablewords, and each edge from the plurality of edges connecting the nodesrepresenting a probabilistic measure of co-occurrence of pairs ofstatistically relevant words in the stream of machine-readable words.20. The system of claim 16, wherein the instructions to generate thesyntax include instructions to generate a connected graph, the connectedgraph including a plurality of nodes and a plurality of edges, each edgefrom the plurality of edges weighted based on a statistical significancescore between each pair of nodes from the plurality of nodes.
 21. Thesystem of claim 16, wherein the memory further stores instructions that,when executed by the processor, cause the processor to: receive a streamof machine-readable words; and update statistics of the co-occurringwords based on the stream of machine-readable words.